Object detection, location, and/or tracking with camera and lighting system

ABSTRACT

A method and system for monitoring objects senses, by an electromagnetic sensor, ambient electromagnetic radiation including at least a particular frequency band reflected from a marker on the object. Data is transmitted about the ambient electromagnetic radiation to a tracking system including at least one memory system and a processor having at least one processor. The processor system determines data representing the marker, the data representing the marker being derived from the data about the ambient electromagnetic radiation. The processor system also determines a location of the object based on the data representing the marker.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority of U.S. Provisional Patent ApplicationNo. 61/464,521, entitled “Object location and tracking with camera andlighting system,” filed Mar. 3, 2011, by Anoo Nathan, which isincorporated herein by reference. U.S. Provisional Patent ApplicationNo. 61/458,978, entitled “Object tracking with camera and light system,”filed Dec. 3, 2010, by Vaidhi Nathan, is also incorporated herein byreference.

FIELD

This specification generally relates to object monitoring systems usinga camera and lighting system.

BACKGROUND

The subject matter discussed in the background section should not beassumed to be prior art merely as a result of its mention in thebackground section. Similarly, a problem and the understanding of thecauses of a problem mentioned in the background section or associatedwith the subject matter of the background section should not be assumedto have been previously recognized in the prior art. The subject matterin the background section may merely represents different approaches,which in and of themselves may also be inventions.

Systems that rely solely on the power of image analysis may suffer fromsignificant error. In order to analyze objects, one must first be ableto identify those objects. Without an effective way to emphasize objectsof interest in a data set, selectively detecting those objects ofinterest can be problematic. This can lead to problems when a largeamount of data must be monitored, and there is no way to determine whichof the data are relevant to a particular inquiry. Without adequateautomation, processes can be too difficult to accomplish with a limitedhuman workforce.

BRIEF DESCRIPTION OF THE FIGURES

In the following drawings like reference numbers are used to refer tolike elements. Although the following figures depict various examples ofthe invention, the invention is not limited to the examples depicted inthe figures.

FIG. 1A shows a block diagram of an embodiment of a system 100 fortracking an object.

FIG. 1B shows a block diagram of an embodiment of a camera 102 a or 102b.

FIG. 1C shows a block diagram of an embodiment of tracking system 106.

FIG. 1D shows a block diagram of an embodiment of memory 146.

FIG. 2 shows a flowchart of an embodiment of a method of monitoringobjects, implemented by control and analysis routines, in whichelectromagnetic radiation can be used to help detect and locate objects.

FIG. 3 illustrates a flowchart of another embodiment of a method ofmonitoring an object implemented by control and analysis routines, inwhich electromagnetic radiation can be used to help detect and locateobjects.

FIGS. 4A-4C Illustrate perspective views of embodiments of the marker.

FIG. 5 Illustrates a perspective view of an embodiment of anelectromagnetic sensor system.

FIGS. 6A-6C illustrate a perspective view of an embodiment of a systemfor monitoring activity.

FIGS. 7A-7C illustrate a perspective view of an embodiment of a systemfor monitoring.

FIGS. 8A-8C illustrate a perspective view of an embodiment of the systemfor monitoring.

DETAILED DESCRIPTION

Although various embodiments of the invention may have been motivated byvarious deficiencies with the prior art, which may be discussed oralluded to in one or more places in the specification, the embodimentsof the invention do not necessarily address any of these deficiencies.In other words, different embodiments of the invention may addressdifferent deficiencies that may be discussed in the specification. Someembodiments may only partially address some deficiencies or just onedeficiency that may be discussed in the specification, and someembodiments may not address any of these deficiencies.

In general, at the beginning of the discussion of each of FIGS. 1-8C isa brief description of each element, which may have no more than thename of each of the elements in the one of FIGS. 1-8C that is beingdiscussed. After the brief description of each element, each element isfurther discussed in numerical order. In general, each of FIGS. 1-8C isdiscussed in numerical order and the elements within FIGS. 1-8C are alsousually discussed in numerical order to facilitate easily locating thediscussion of a particular element. Nonetheless, there is no onelocation where all of the information of any element of FIGS. 1-8C isnecessarily located. Unique information about any particular element orany other aspect of any of FIGS. 1-8C may be found in, or implied by,any part of the specification.

This application incorporates herein by reference in U.S. Pat. No.8,075,499, by Vaidhi Nathan et al., patented Dec. 13, 2011 (whichteaches the detection of abnormal behavior, such as epilepsy), U.S.Provisional Patent Application Ser. No. 60/898,472, filed Jan. 30, 2007,Provisional Patent Application Ser. No. 60/898,603, filed Jan. 30, 2007,Utility patent application Ser. No. 12/011,705, filed Jan. 28, 2008, andUtility patent application Ser. No. 12/072,186 (which teaches stitchingimages together to form a single image). Provisional Patent ApplicationSer. No. 60/898,472, filed Jan. 30, 2007, Provisional Patent ApplicationSer. No. 60/898,603, Utility patent application Ser. No. 12/011,705include teachings that related to background and foreground extraction,which are useable in conjunction with the methods taught herein.

FIG. 1A shows a block diagram of an embodiment of a system 100 fortracking an object. System 100 may include cameras 102 a and 102 b,optional bus 104, tracking system 106, object 108, marker 110, and theelectromagnetic radiation emitting system 112. In other embodiments, thesystem 100 may not have all of the elements listed and/or may have otherelements in addition to or instead those listed.

Cameras 102 a and 102 b capture images of the object and may track theobject as the object moves. Although only two cameras are shown, theremay be any number of cameras in system 100. Any place where cameras 102a and/or 102 b are mentioned, any number of cameras may be substituted.

The cameras 102 a and 102 b may be any devices configured to captureimages including, for example, any one of, some of, any combination of,or all of a fixed camera, a moving camera, a high definition camera, ananalog camera with digital converter, a panoramic camera, a highresolution camera, an infra-red camera, a pan-tilt-zoom camera, acommercial off the shelf camera, a web-camera, a camera on a mobileelectronic device, a satellite camera, and a microwave camera, forexample.

In one embodiment, the cameras 102 a and 102 b may be pan-tilt-zoomcameras that determine their direction of movement based on a specifiedangle in a specified direction, based on a specified speed in aspecified direction, and then quickly provide responses to commands evenwhile moving at a particular speed. Experimentally, it is generallyaccepted that position and velocity measurements provide smoother, moreaccurate motions for tracking.

The cameras 102 a and 102 b may be configured to work in concert withone another, creating complete, seamless fields of view throughout anarea.

The cameras 102 a and 102 b may have different roles. In ananti-terrorist setting, the cameras 102 a and 102 b may be used toprovide digital images of suspicious objects and track the location ofthose images throughout a sensitive terrorist target area. The cameras102 a and 102 b may be used in a hospital to track patients moving fromone area to another, to monitor unauthorized access to controlledsubstances by patients, to monitor activity of an epilepsy patient inorder to determine if the epilepsy patient is experiencing an epilepticseizure.

The cameras 102 a and 102 b may move or may be fixed. Those that movemay move from point to point; smoothly and continuously; in a wait,move, stop, and move again pattern; move when an object approaches thelimit of a viewing angle; or keep the object at center at all times.

The optional communications bus 104 communicatively couples, cameras 102a, 102 b, and the rest of the tracking system. The optionalcommunications bus may communicate with the cameras 102 a and 102 b by awired or wireless connection. The wireless connection can be Wi-Fi,Wi-Max, 4G, 3G, or Bluetooth, for example. The wired connection can beUSB, firewire, SATA, or HDMI, for example.

Tracking system 106 controls the positioning, focus, and/or zoom of thecameras 102 a and 102 b, analyzes the data received from cameras 102 aand 102 b. The tracking system 106 may be a programmed computer and/or aspecialized controller. The tracking system is more extensivelydiscussed in FIG. 1C.

Object 108 is the object being tracked which could be any object, whichmay be a living organism or an inanimate object. As one example, object108 may be a patient, such as a person with epilepsy. As an example ofan inanimate object, object 108 may be jewelry or another object ofvalue. Object 108 is tracked by cameras 102 a and 102 b under thecontrol of tracking system 106.

Marker 110 may be a piece of material that reflects and/or emits aparticular relatively narrow band of light. For example, the band oflight may be narrow enough that for engineering purposes, the band oflight is considered as a single wavelength. Cameras 102 a and 102 bdetect the wavelength of light transmitted from marker 110, trackingsystem 106 analyzes the data detected and determines the signals to sendto cameras 102 a and 102 b to track to reposition, refocus, and/oradjust the zoom of the cameras 102 a and 102 b to track object 108. Inan embodiment, cameras 102 a and 102 b or tracking system 106 may storea log of the position, significant events (e.g., an event that is likelya seizure or other abnormal behavior), and/or the orientation of object108.

The marker 110 may include, for example, any one of, some of, anycombination of, or all of a paint, an ink, an adhesive substance, aspray on substance, a balm, clothing such as shoes or a shirt, a hat, atag, a card, a patch, a substance injectable into living or nonlivingthings, and the like. The marker 110 may consist of multiple items inorder to make it easier to detect by an object detector of the analysisand control routines 147 (which will be discussed in FIG. 1C). The bandof light which the marker 110 may reflect or emit may be detected by thecameras 102 a and 102 b or by an optional electromagnetic sensor system.

In another embodiment, the marker 110 may emit and/or reflect all bandsof Electromagnetic Radiation (EMR), any part of a band of EMR, anycombination of bands of electromagnetic radiation, and the like. Forexample, the marker 110 may be used to emit and/or reflect the EMR bandof a particular type of EMR, for instance, by solely detecting the bandbetween 790 terahertz to 400 terahertz, representing only the visuallight spectrum. Alternatively, the marker 110 may selectively reflectand/or emit a small section of the x-ray band, such as 20 exahertz to 5exahertz. Alternatively, the marker 110 may reflect and/or emit the bandof radiation represented by segments of a few consecutive bands, variousdifferent bands within different spectra, The marker 110 reflects and/oremits a particular band of EMR, which an optional electromagnetic sensorsystem may detect, and the electromagnetic sensor may transmit dataabout the ambient electromagnetic radiation to be processed in theanalysis and control routines 147 (FIG. 1C).

The marker 110 may be configured to emit a particular band of light orother EMR at specific intervals using a local, generalized, or wirelesspower source. In order to save power, the marker need not transmit itssignal continuously. For instance, the marker 110 could transmit EMR fora hundred milliseconds once every second. The marker may also emit onlya particular intensity of EMR depending on the expectation of backgroundEMR in the band that the electromagnetic sensor is configured to detect.

The marker 110 may be configured to both reflect and emit EMR in aparticular band. For instance, the marker 110 may detect the level ofambient relevant EMR and determine that there is insufficient EMR torely on a marker's 110 reflection. This may trigger the marker 110 touse a power source to emit ER. Alternatively, the marker 110 may beconfigured to consistently reflect and emit either the same or differentbands.

The marker 110 may also be specially shaped in order to allow for betterdetection. For instance, the marker may be shaped like a bar, an “x,” orlike a crosshair in order to allow more effective execution of theanalysis and control routines 147 (FIG. 1C) in order to facilitatelocation and/or identification. The shape of the marker 110 may also beused to determine the orientation of an object using the analysis andcontrol routines 147. For the purposes of this specification, the marker110 may be called either “marker” or “tag.”

The Electromagnetic Radiation Emission System (“ERES”) 112 may include,for example, any one of, some of, any combination of, or all of atransmitter, incandescent substance, a prism, a lamp, a microwaveemitter, a radio, any other electromagnetic radiation emitting device,and the like. An ERES 112 may be necessary because the intensity of aparticular band of local ambient EMR may be too weak for reliabledetection. The ERES 112 may be configured to emit EMR consistently, atdifferent intensities, when triggered by the analysis and controlroutines 147 (FIG. 1C) or by any other system, periodically, or inresponse to an alert, for example.

The ERES 112 may emit all bands of EMR, any part of a band of EMR, anycombination of bands of electromagnetic radiation, and the like. Forexample, the ERES 112 may be used to emit the EMR band of a particulartype of EMR, for instance, by solely detecting the band between 790terahertz to 400 terahertz, representing only the visual light spectrum.Alternatively, the ERES 112 may selectively emit a small section of thex-ray band, such as 20 exahertz to 5 exahertz, for example.

The ambient EMR may contain several different elements, and the ERES 112may be used to emit a particular band of EMR in order to accentuate thereflectivity of the particular band of EMR. The ERES may not benecessary if the reflectivity of the marker can make sufficient use ofthe ambient electromagnetic radiation or if the marker 110 is itself anemission device.

FIG. 1B shows a block diagram of an embodiment of a camera 102 a or 102b. Camera 102 a or 102 b may include output system 122, input system124, memory system 126 a having zoom and focus drivers 126 b,positioning drivers 126 c, and/or sensor interface 126 d, camera 102 aor 102 b may also include processor system 128, input/output 130,optional electromagnetic sensor 134, motors for zoom and focus 136,motor for positioning 138, lens system 139 a, image sensor 139 b,transmission 139 c, and optional optical filter 139 e. In otherembodiment, the camera 102 a or 102 b may not have all of the elementslisted and/or may have other elements in addition to or instead of thoselisted.

Cameras 102 a or 102 b track an object. Output system 122 may include anoutput information panel including information about the focus, zoom,f-stop, shutter speed, or the amount of time the frame that is exposed,and/or the orientation of the cameras 102 a and 102 b. Output 122 mayalso include an interface for sending image data and control informationto tracking system 106 (FIG. 1A). Output system 122 may communicate withremovable machine readable media in order to store information on themachine readable media. Output system may also be configured tocommunicate with an optional electromagnetic radiation emission devicein order to control output of electromagnetic radiation.

Input system 124 may include a control panel for adjusting the focus,zoom, and/or position of cameras 102 a and 102 b. Input 124 may includean interface receiving control signals controlling the position, zoom,and focus of cameras 102 a and 102 b from tracking system 106.

Memory system 126 a may store applications containing algorithms thatcontrol cameras 102 a and 102 b and/or data recorded by cameras 102 aand 102 b. The memory system 126 a may also store zoom and focus drivers126 b, positioning drivers 126 c, and the sensor interface, for example.

Zoom and focus drivers 126 b may include algorithms (machineinstructions), which when run by the processor system generate controlsignals that control the motors, which adjust the positions of thelenses and thereby control the zoom and focus of the lenses.

Positioning drivers 126 c may include algorithms (machine instructions),which when run by the processor system generate control signals thatcontrol the motors that adjust the position of cameras 102 a and a 102b, thereby controlling the direction in which cameras 102 a and 102 bare pointed. The positioning drivers 126 c may instruct the cameras 102a and 102 b to move from point to point; smoothly and continuously; in await, move, stop, and move again pattern; move when an object approachesthe limit of a viewing angle; or keep the object at center at all times.

Sensor interface 126 d may be an algorithm for storing and/orinterpreting data from an image sensor and/or from a light sensor thatsenses the narrow band of light transmitted from marker 110.

Processor system 128 runs the algorithms stored in memory 126 a, such aszoom and focus drivers 126 b, positioning drivers 126 c, and sensorinterface 126 d. Consequently, processor 128 determines the position,focus, and zoom of the camera by sending control signals to the motorsthat control the transmission and the lens positions. Input/output 130may perform any of the functions of input 124 and/or output 126.

The optional electromagnetic sensor 134 may detect the narrow band oflight transmitted from marker 110. Alternatively, the optionalelectromagnetic sensor 134 may detect a variety of bands ofelectromagnetic radiation (“EMR”) outside of the visible light spectrum.

The motors for zoom and focus 136 control the positioning of the lensesto thereby control the zoom and focus of the lenses system. The motorfor positioning 138 positions cameras 102 a and 102 b (FIG. 1A). Lenssystem 139 a may be the lenses that focus the incoming light onto animage sensor, such as a charge couple device (CCD). Image sensor 139 bsenses the light from the lens system 139 a and sends the data to memory126 a. Transmission 139 c is the gear system connected to motor used toposition cameras 102 a and 102 b. The analysis and control routines 147(FIG. 1C) may include algorithms to alternate focal lengths whencapturing images in order to determine the depth of an object.

The optional optical filter 139 e may be used to limit the amount oflight from certain spectra that enters a camera. For instance, in oneembodiment, the optional optical filter 139 e may limit by 50% of theextent to which all visible light, except green light, enters the lenssystem 139 a of the cameras 102 a and 102 b (FIG. 1A). In doing so,anything green will appear brighter, because the optional optical filter139 e filtered out other colors to some extent.

Colors represent merely the visible light element of the electromagneticspectrum. In an alternative embodiment, the optional optical filter 139e may filter out other electromagnetic radiation, for instance a part ofthe infra-redspectrum.

FIG. 1C shows a block diagram of an embodiment of tracking system 106.Tracking system 106 may include output system 142, input system 144,memory system 146, control and analysis algorithms 147, processor system148, communications bus 152 and input/output system 154. In otherembodiments, the tracking system 106 may not have all of the elementslisted and/or may have other elements in addition to or instead of thoselisted.

The output system 142 may include any one of, some of, any combinationof, or all of an electromagnetic radiation emitting system, a displaysystem, a printer system, a speaker system, a connection or interfacesystem to a sound system, an interface system to peripheral devicesand/or a connection and/or a interface system to a computer system,intranet, and/or internet, and the like. Output system 142 may include amonitor and/or other output device. Output system 142 may include aninterface for sending output signals to cameras 102 a and 102 b or theoptional communications bus 104, indicating the position, focus, zoomand aperture for cameras 102 a and 102 b (FIG. 1A).

The input/output system 154 may be configured to communicate databetween the electromagnetic sensor and the hardware system. For instancethe input/output system may relay data about the ambient electromagneticradiation to the hardware system. The input/output 154 system mayfacilitate communications from the image capture system, the networkinterface system, the electromagnetic sensor system, and theelectromagnetic radiation emission system 112 to the hardware system.

The input system 144 may include any of, some of, any combination of, orall of a keyboard system, an interface to receive image and sensor datafrom cameras 102 and 102 b, a mouse system, a track ball system, a trackpad system, buttons on a handheld system, a scanner system, a microphonesystem, a touchpad system, and/or a connection and/or interface systemto a computer system, intranet, and/or internet (e.g., IrDA, USB), andthe like. Input system 144 may receive information about the currentzoom, focus, aperture, and position of cameras 102 a and 102 b (FIG.1A). Input system may also receive all image and sensor data fromcameras 102 a and 102 b as well as the optional electromagnetic sensor134 (FIG. 1B).

Memory system 146 may store algorithms for analyzing data received fromcamera 102 a and 102 b (FIG. 1A) and determining the position andorientation for the object 108 and for determining the position,aperture, focus, and zoom appropriate for tracking object 108 (FIG. 1A).The memory system 146 may include, for example, any one of, some of, anycombination of, or all of a long term storage system, such as a harddrive; a short term storage system, such as a random access memory; aremovable storage system such as a disk drive, floppy drive or aremovable drive; and/or flash memory. The memory system 146 may includeone or more machine readable media that may store a variety of differenttypes of information. The term machine-readable media may be used torefer to any medium capable of carrying information that is readable bya machine. One example of a machine-readable medium is acomputer-readable medium.

The memory system 106 may be configured to store the applicationsnecessary to provide instructions to the analysis and control routines147 in order to accomplish the computations mentioned above in thediscussion of analysis and control routines 147. The memory system 146may also store variables, intermediates, results, constants, and thelike necessary to execute the analysis and control routines. The memorysystem 146 may store a log of events representing activity of a marker.The memory system 146 may also be configured to store a databasecontaining image capture data fitting certain criteria, for instance,motion of a patient in a hospital, motion of passengers in an airport,trespassing, removal of an object from a particular place, and the like.

The memory system 146 may include at least one memory device and maystore analysis and control routines 147. Analysis and control routines147 are the algorithms for analyzing data received from camera 102 a and102 b and determining the position and orientation for the object 108and for determining the position, aperture, focus, and zoom appropriatefor tracking object 108 (FIG. 1A) Embodiments of the analysis andcontrol routines 147 are explained in greater detail in FIG. 1D.

Processor system 148 runs the algorithms and analysis control routines147 stored in memory 146. Consequently, processor 148 analyzes datareceived from camera 102 a and 102 b for determining the position andorientation of the object 108 and for determining the position,aperture, focus, and zoom appropriate for tracking object 108 (FIG. 1A).The processor system 148 transmits data to adjust the position, focus,aperture, zoom of cameras 102 a and 102 b. Input/output 154 may performany of the functions of input 142 and/or output 144.

The processor system 148 may include any one of, some of, anycombination of, or all of multiple parallel processors, a singleprocessor, a system of processors having one or more central processors,a logic circuit and/or one or more specialized processors dedicated tospecific tasks.

FIG. 1D shows a block diagram of an embodiment of memory 146. Memory 146may include analysis and control routines 147, which in turn may includepreprocessor 160, background subtraction 162, object detector 164,object validate 166, object tracker 168. In other embodiments, thememory 146 may not have all of the elements listed and/or may have otherelements in addition to or instead those listed.

Analysis and control routines 147 may be executed by an embodiment ofthe processor system 148. The preprocessor may preprocess the sensor orvideo data to make the data suitable for further processing andanalysis. The preprocessing involves steps such as noise reduction andcontrast enhancement.

Background subtraction 162 may model the background conditions of thescene, thereby accentuating the non-background pixels in the scene atthe time of processing. The background model can be static or dynamicand adaptive. Static models are easier to compute, have lowcomputational complexity, but are applicably in those cases where sceneconditions do not change with time.

A variety of background modeling algorithms can be used for determiningthe background, such as a fixed-threshold approach (e.g., if themovement between frames is below a fixed threshold, then the feature inquestions is grouped as background), or algorithms like Gaussian mixturemodeling, kernel density estimators, mean-shift filtering, and Kalmanfiltering, for example. The approach taken in the embodiment of FIG. 1Daccounts for the feedback from the object tracker to determine whichpixels in the scene are currently part of the foreground, and hencesuppresses the updating of the background model in those regions.

In each of the Gaussian mixture modeling, kernel density estimator, meanshift filtering, and Kalman filtering a density function is computedthat represents the background of the image. Gaussian mixture modelingrepresents a parametric probability density function as a weighted sumof Gaussian component densities. A Gaussian mixture model is a weightedsum of M component Gaussian densities as given by the equation,p(x|λ)=Σ_(i=1) ^(M) w _(i) g(x|μ _(i),Σ_(i)),

where x is a D-dimensional continuous-valued data vector (i.e.measurement or features), are the component Gaussian densities andw_(i), i=1, . . . , M, are the mixture weights, and G(x|μ_(i), Σ_(i)),i=M are the component Gaussian densities. Each component density is aD-variate Gaussian function of the form,

${g\left( {\left. x \middle| \mu_{i} \right.,\sum\limits_{i}} \right)} = {\frac{1}{\left( {2\pi} \right)^{D/2}{\sum\limits_{i}}^{1/2}}{\mathbb{e}}^{{- \frac{1}{2}}{({x - \mu_{i}})}{\sum\limits_{i}^{- 1}{({x - \mu_{i}})}}}}$

with mean vector μ_(i) and covariance matrix Σ_(i). The mixture weightssatisfy the constraint thatΣ_(i=1) ^(M) w _(i)=1.

The complete Gaussian mixture model is parameterized by the meanvectors, covariance matrices and mixture weights from all componentdensities. The mean vectors, covariance matrices and mixture weightsparameters are collectively represented by the notation,λ={w _(i),μ_(i),Σ_(i) } i=1, . . . ,M.

There are several variants on the GMM. The covariance matrices, Σ_(i),can be full rank or constrained to be diagonal. Additionally, parameterscan be shared, or tied, among the Gaussian components, such as having acommon covariance matrix for all components, the choice of modelconfiguration (number of components, full or diagonal covariancematrices, and parameter tying) is often determined by the amount of dataavailable for estimating the GMM parameters and how the GMM is used in aparticular application. It is also important to note that because thecomponent Gaussian are acting together to model the overall featuredensity, full covariance matrices are not necessary even if the featuresare not statistically independent. The linear combination of diagonalcovariance basis Gaussians is capable of modeling the correlationsbetween feature vector elements.

Kernel density estimation is a non-parametric method, unlike theGaussian mixture model. Kernel density estimation attempts to estimatethe probability density function of an unknown variable. In thefollowing equation, (x₁, . . . , x_(n)) may be from a data sample withan undetermined density, f, K(x) may be a symmetric kernel, h may bebandwidth, K_(h)(x) may be a scaled kernel defined as K_(h)(x)=1/hK(x/h),

${f_{h}(x)} = {{\frac{1}{n}{\sum\limits_{i = 1}^{n}{K_{h}\left( {x - x_{i}} \right)}}} = {\frac{1}{nh}{\sum\limits_{i = 1}^{n}{K\left( \frac{x - x_{i}}{h} \right)}}}}$

A common method for estimating the bandwidth, h is the followingfunction defining the normal distribution approximation,

${h \approx {1.06\;\sigma\; n^{- \frac{1}{5}}}},$where σ is the standard deviation of the samples.

Bandwidth may be more precisely estimated using the mean integratedsquare error. This may be approximated using the following equation:MISE(h)=E∫((f _(h)(x)−f(x))² dx

Where f_(h)(x) is the kernel density estimator dampened by bandwidth h.

Mean shift filtering is a non parametric, mode-seeking algorithm used tolocate density function maxima. For the kernel function, K(x_(i)−x), theweighted mean of the density in the window determined by K(x) may beexpressed as:

${m(x)} = \frac{{\Sigma\; x_{i}} \in {{N(x)}{K\left( {x_{i} - x} \right)}x_{i}}}{{\Sigma\; x_{i}} \in {{N(x)}{K\left( {x_{i} - x} \right)}}}$

Where N(x) is the neighborhood of x, a set of points for which K(x) doesnot equal zero. The equation is repeated iteratively until m(x)converges with x.

The Kalman filter determines instances in which the K value representingthe density function is better represented by a prediction equationusing prior measurements than by the current measured value. In doingso, it creates smoother transitions between data and eliminates pointsthat are more likely to be noise. The algorithm may determine whetherthe measurement error covariance is low, making the actual measuredvalue more trustworthy and hence accepted. The algorithm mayalternatively determine that the a priori estimate is low enough thatthe actual measurement is less trustworthy than the estimated value.

The object detector 164 may take as an input the location of foregroundpixels in the image (represented as a two tone image), and analyze themto detect objects in a particular scene. It may begin by delineatingcontours in the two-tone image, which generally represent the connectedcomponents. The resulting contours may be treated as candidate objectsand can be further analyzed to filter out irrelevant objects usingcriteria such as minimum and maximum size and history information suchas the presence/absence of tracked objects in the vicinity. In anembodiment in which the camera is fixed, motion-based object detection,such as optical flow or simple frame differencing, may also be used todetect and/or track the location of the object of interest.

The object detector 164 can determine the location of the object byusing a variety of methods including a stereo image depth computation,three-dimensional orientation computing, object pose/orientationcomputation or the like.

One Primary Reference Sensor & Multiple PTZ Cameras (for Multiple TAGS)

In this scenario, there is 1 primary sensor (reference camera) and 2 or3 or multiple following cameras.

The reference camera/sensor, identifies one or more objects or people.Since each person or object may need to be tracked, then each object orperson needs its own PTZ/moving camera assigned to it. Hence the systemwill assign each moving object/person to each of the moving/PTZ camerato track. For example, a room may have 2 people in it and moving aroundthat may need to be tracked. Each person will have a TAG (which may bethe same or different). The primary/reference sensor, detects the 2 tagsand identifies the objects and locations to track. Then the system willassign Object #1 to Moving-PTZ-camera #1 to track. Then assign the2^(nd) object/person to 2^(nd) camera to track. Then each camera tracksits own object/target. The primary sensor provides the current locationinformation. This can be extended to 3 or 4 or many following cameras.

This is similar to a 1-many relationship for ONE primary-referencesensor to MANY following secondary PTZ cameras.

If the TAG is SAME, then the assignment may be random and the mixing ofpeople may be possible sometime. To avoid mixing people, system will usethe location, direction, speed, size, brightness, intensity, histogramand other imaging parameters to decide and differentiate and identifythe right target or object. Then follow it.

If the TAGS are DIFFERENT, meaning the tag size, shape, intensity, aredifferent, then the TAG itself can be used to distinguish and identifythe right person and not mix the person ID. With different TAGs, correctID for a person can be maintained. Hence the camera will follow theright person

Multiple PTZ following cameras can be used in both of the followingcases:

Case 1: 2 or more people and each camera is following one person

Case 2: 1 Person that moves between different areas or zones. Camera 1tracks zone/area 1; and camera #2 tracks Zone/area 2.

Panorama/360/Wide-Angle Camera with HD or Higher Resolution

There are new types of cameras that see full 360 or 270 or wide angleviews. They can also be HD (highDEF) or high resolution or mega-pixelcameras. Typically they have 1920×1080 or mega pixels can be 4000×3000or different high resolutions. Some have 5M or 8M or 10M pixels. (8Mpixels will have 8 million pixels). With these, these PTZ cameras can bereplaced. These fixed Panorama or HD/HighRes cameras will be used. Thenthe object location and position is identified from the primary orreference camera. Then the object location and video is “cut out” orextracted or a “sub-window” is extracted from this large mega pixelcamera. Then this video is streamed out from the moving sliding window.For example, the output may be SD (640×480). The object location can beidentified and a window of 640×480 (or a similar window) can be used toplace and extract the video images from the camera. As the target moves,the window location and size is changed/repositioned. This video oflower resolution is output, similar to the PTZ camera. This is a newinnovative alternative to PTZ cameras, where a fixed megapixel cameraacts like a digital or electronic PTZ in which the moving window isincluded with an output video.

With this, a 1 mega pixel camera can act as (or accommodate severalsmaller windows or PTZ cameras, say 4 or 6). A real time video output ismade from each of these smaller windows (as though they are a PTZ movingcamera), but actually the video is extracted and sent from a fixedHD/high resolution camera.

This can be cost effective in some areas and reduce the number ofcameras.

When using a panorama and 360/270 deg cameras, the image is warped andthe images need to be de-warped and made planner and flat so that theimages are viewable. De-warping is part of this step before the normalvideo is output

A principle that may be used to obtain a stereo image is triangulation.The three-dimensional location of a world-point projected onto imageplanes can be computed by finding the intersection of the two linespassing through the center of the image and the projection of the pointin each image. In order to find the intersection of the lines passingthrough the projections, first a correspondence problem needs to besolved that determines the correspondence of the point in the image tothe objects detected. Solving the correspondence problems can beaccomplished by a variety of methods, such as minimizing the matchingerror based on some local measurement (color, intensity, texture etc.),multi-scale edge matching, dynamic programming, relaxation labeling,graph cutting, and/or other techniques for determining thecorrespondence.

Once the correspondence problem has been solved, a three-dimensionalreconstruction can be made by utilizing the following equation:Z=(B*f)/d

Here, ‘B’ represents the distance between the centers of projection,known as the baseline, ‘f’ represents the camera focal length, and ‘d’represents the distance between the corresponding points when the twoimages are superimposed, known as the disparity, d, which can beexpressed as d=(x1−x2).

In an embodiment, B may be the distance between cameras. The point P mayappear in different locations in the two Stereo image capture devices.For example, Point P will appear at a location X1 in a first imagecapture device and a location X2 in a second sensor or image capturedevice. Using these locations X1 and X2, the disparity distance iscomputed, which is also called off-set or disparity distance. From thedisparity distance, the mathematical triangulation can be computed andobject or point depth Z can be obtained.

It should be noted that the above formulations have been provided for analigned stereo rig. For unconstrained stereo rigs, a prior step ofrectification may be required. In this case, known sensor or cameracalibrations, although not required, simplify the rectification process.Further, rectification is also strictly not required, but it makessolving the correspondence problem much less computationally expensive.

For general stereo problems, more than 2 perspectives can be utilizedfor more robust correspondence estimation and for resolving ambiguities.For example, in a three image capture system, the correspondence can beestimated between the first and second image capture devices, the 3Dpoint reconstructed, and then projected again into the third imagecapture device. If a matching point can be found near the projectionpoint on the third image capture device, then the correspondencesolution may be accepted or rejected.

The accuracy of three-dimensional reconstruction may be heavilydependent upon the accuracy of point correspondence. As mentioned above,approaches exist for estimating good correspondence, and these can befurther improved by employing more than 2 views. However, the downsideof such approaches is heavy computational cost, expensive hardware andin most cases the processing is not performed in real-time.

In an embodiment of the current method of tracking and locating objectsthe correspondence problem can be solved, without requiringcomputationally intensive algorithms, expensive hardware or more thantwo cameras (and therefore can be solved in real time). In anembodiment, the object-of-interest is uniquely defined and can be easilyseparated from other elements of the scene based on visual appearance.Tracking and locating the object without intensive computing is possibleby employing reflective light technology such that the spectralsignature (intensity, or any color band, or any particular range of theEM spectrum) received for the object-of-interest is unique and can beeasily differentiated from the background. Any range of the EM spectrumwaves can be used by the source and sensor. For example, the range ofthe EM spectrum that may be used by the source and sensor may bevisible, IR, UV, X-ray and thermal wavelength sources and correspondingsensors to observe that wavelength. Further, for accurate localizationof the feature point used for correspondence, the shape of the objectmay be designed such that the shape offers uniquely distinguishablefeature points. For example, the object-of-interest may be cross-shapedand the intersection point of the arms of the cross can be easilydetected and used as a feature point for the correspondence problem.Similarly, other shapes and configurations are possible.

Regarding the problem of computing the object's orientation with respectto the camera, the problem can be formally stated as: given a set ofpoints defined in the object-centered frame, and their projections ontothe image plane, determine the transformation between theobject-centered and image-centered frame. The transformation hereincludes rotation (R) and translation (t) components.

Various methods have been proposed in the literature for solving thetranslation problem. Assuming that the object model is known (that is,the geometric configuration of the features on the object is known), themethods may be categorized into at least 2 categories. If the number ofpoint correspondences is limited, a closed-form solution may be used.Closed form solutions may be used when there are 3 points, 3 lines, 4coplanar points, and/or 4 general points, for example. When the numberof point correspondences is greater than 6, iterative numericalsolutions may be used to determine orientation. The iterative methodsalthough more robust, suffer from the drawback that if an approximateinitial pose is not known well enough, the number of iterations may benumerous and/or the problem may be unsolvable. In the following, webriefly describe a method (called POSIT) that is iterative but does notrequire an initial pose. The equations relating the perspectiveprojections are:

${P_{0}{P_{i} \cdot \frac{f}{Z_{0}}}i} = {{x_{i}\left( {1 + ɛ_{i}} \right)} - x_{0}}$${P_{0}{P_{i} \cdot \frac{f}{Z_{0}}}j} = {{y_{i}\left( {1 + ɛ_{i}} \right)} - y_{0}}$$ɛ_{i} = {\frac{1}{Z_{0}}P_{0}{P_{i} \cdot k}}$

Here P₀ is a reference point on the object (relative to which the objectgeometry is defined), P_(i) is any arbitrary feature point on theobject, f is the focal length of the camera, Z₀ is the distance of P₀from camera along the z-axis, (x₀, y₀) are the coordinates of the imageof the point P₀ on the image plane, (x_(i), y_(i)) are the coordinatesof the image of the point P_(i) on the image plane, and i, j, k are therow vectors of the rotation matrix R given by

$R_{3 \times 3} = {\begin{pmatrix}i \\j \\k\end{pmatrix}.}$

Note that only two rows of the rotation matrix are independent. Forexample, k=i×j. It may be desired to find i and j which will give thecomplete pose of the object. In the POSIT algorithm, the above set ofnon-linear equations is solved linearly and iteratively by assigning afixed value to ε_(i), starting with ε_(i)=0, and then iterativelycorrecting it using the value of ε_(i) computed by solving the equationsin the previous iteration step. In an embodiment, a feature of thetechnique of solving for orientation is that the above set of equationsare solved linearly and directly (without any iterating, the depthcomputation is performed to solve for Z), the above set of equationsbecome linear with six unknowns, and hence locating a three featurepoints correspondence is sufficient to solve for the object poseaccurately. Some advantages of embodiment in which the POSIT equationsare solved using Z for the depth computation are, there are fewerfeature point correspondences required; the computation is moreefficient because no iterations are required; and the equations yield amore accurate solution because the equations are solved in a closedform.

As an option, a secondary Pan Tilt Zoom (PTZ) or moving image capturedevice can be used to zoom in and track closely the primary objects. APTZ camera is a closed-circuit television image capture device withremote directional and zoom control. Any camera with a remote pan, tilt,and zoom control may be substituted.

An engine may be used to accomplish one of, both of, or at least twothings: the engine maps the (X, Y) location of the object being trackedinto a (P, T, Z) value. The mapping may done with a calibration map. Thecalibration map is scene-dependent but is a one-time configuration step.Then the PTZ camera may be moved so that the error between the desired(or measured) PTZ values and the current (or theoretically computed) PTZvalues may be driven to zero as instructions are given to the camera tomove using speed instructions. The camera may be moved using speedinstructions as opposed to position instructions because moving usingspeed instructions results in a smoother movement of the PTZ camera.Proportional, Integral, and Derivative (P I D) control may be used toaccomplish to zero the difference between the desired PTZ settings andthe current PTZ settings.

The object detector 164 may use the above methods to locate or identifyliving beings or non-living objects such as boxes, vehicles, or parcelsin production or shipping, and/or other objects. The tags may identifythe location and orientation of these objects or boxes or any things inspace.

The object validator 166 may be used to select objects that meetparticular appearance criteria and filter out the rest of the image. Adistinguishing feature could be a specific shape, color, intensitypattern or any combination thereof, which provides a visual cue fordetecting the object. In an embodiment, a combination of specific shapeand intensity patterns are used. The object validator 166 may alsorecord the history of each tracked object, and maintain information,such as how the features of the object have evolved over time. Trackingthe history of each object is especially useful in suppressingoccasional false objects that otherwise seem to meet the visualappearance criteria for objects of interest.

The object tracker 168 may be responsible for tracking the detectedobjects temporally across frames. There are various methods for objecttracking that are used in different embodiments. Different types offeature representations could be used such as color, intensity, texture,Scale Invariant Feature Transform (SIFT) features, silhouette, and/orother feature representations. Once the features have been computed, acorrespondence may be established between these features temporallyacross adjacent frames. This can be done using deterministic orprobabilistic methods. For example, optical flow algorithms can be usedto track feature points based on intensity. Also, motion constraintssuch as proximity, small velocity changes, rigidity, etc. can beimposed. Template based methods such as mean-shift tracker can also beused.

The object tracker 168 may also cause a pan/tilt/zoom camera to follow apatient to get three coordinates of location data or each of the pantilt and zoom measurements. The object tracker 168 may instruct thecamera to follow an object to determine its current position. This maybe useful to track whether a living being or object remains in alocation it is supposed to be or strays into a location where the livingbeing or object should not be.

Alternatively, the object tracker 168 may also execute instructionsstored in an application on the memory system to track unusual movementof a being for medical reasons. For instance, the tracker may detect astrange motion of a person that gives the appearance the person ishaving an emergency medical event, like a heart attack. The tracker maydetect that a person is bending over in an area where people areunlikely to be bending over in order to notify an official that theperson is more likely to cause a disturbance somewhere, for instance inan airport. The tracker may also track object motion for strangemovements, not only to determine if the object is safe, but to determinelocal seismic activity.

The object tracker 168 may follow instructions of an application storedon the memory that includes detection software in order to track objectsmoving from one image capture device's view to another image capturedevice's field of view. Object tracker 168 may accomplish this bydetecting where the object leaves the frame of one image capture deviceand where the object may return or enter the frame of another imagecapture device. This may be accomplished by the system determining thearea where the object left, and try to reacquiring or recapturing theobject.

Alternatively, the object tracker 168 may instruct a camera to await thereentry of the object that left the view of a particular image capturedevice in order to detect the object. If for any reason, the objectdisappears, the system may wait for the object to reappear.

The object tracker 168 may be able to signal to a particular detectionand location image capture device to signal another detection andlocation image capture device to track a particular object.Alternatively, the analysis and control routines 147 may be able tosignal the exchange of an object from the field of view of one trackingcamera to the field of view of another tracking camera. The camerasthemselves may trigger one another to follow an object as well. Locationand identification cameras may be able to trigger the tracking cameras,other location and identification cameras, or other cameras. Thetriggered cameras could also trigger those cameras.

The object tracker 168 may be able to use data from a number ofdifferent image capture devices. For instance, the system may include afixed camera with a large viewing angle and a pan-tilt-zoom camera witha smaller angle. Using both cameras simultaneously could allow a user alarge view overall using the fixed camera while the pan tilt zoom cameracould focus on a specific area. The fixed camera could be used to detectand locate objects. The pan-tilt-zoom camera could be used to track themovement of the objects. Tracking cameras can be used to zoom in andclosely follow objects. In another embodiment, this two-camera systemcould use two fixed cameras. The primary camera could still be used todetect and locate objects. The second camera, instead of physicallymoving in order to zoom, the camera itself could yield a portion of theimage that correlates with the object detected. The camera can outputvideo data, which represent an image of a closer, focused view of theobject. As such, it would have the effect of providing a pan-tilt-zoomimage. This would allow it to track in much the same manner as apan-tilt-zoom camera. For instance, the secondary camera could have anative resolution of 1920×1080. The video output could focus on a640×480 video within the camera's native resolution, allowing the outputto be better focused on the object.

The object tracker 168 may be able to calculate the position andvelocity of a person. The position and velocity data can be used to tryto predict the path of a detected object and to determine its likelydestinations. For instance, a person moving at six meters per second isunlikely to stop and turn at a doorway five inches in front of him, buthe may still likely enter a doorway fifteen feet away.

FIG. 2 shows a flowchart of an embodiment of a method 200 of monitoringobjects, implemented by control and analysis routines 147, in whichelectromagnetic radiation can be used to help detect and locate objects.In this embodiment, the sensor system may detect a particular band ofelectromagnetic radiation and use the particular band of radiation tolocate a marker on an object. In step 202, an electromagnetic sensorsystem may detect electromagnetic radiation using at least oneelectromagnetic sensor. The electromagnetic sensor system may be used todetect all bands of electromagnetic radiation (“EMR”), any part of aband of EMR, or any combination of bands of electromagnetic radiation,for example. EMR can be of many different types depending on itsfrequency. In an embodiment, the electromagnetic sensor senses a narrowband of radiation.

The electromagnetic sensor system may be used to detect the EMR band ofa particular type of EMR, for instance, by solely detecting the bandbetween 790 terahertz to 400 terahertz, representing only the visuallight spectrum. Alternatively the bandwidth may only be 10 terahertz.Alternatively, the electromagnetic sensor system may selectively detecta small section of the x-ray band, such as 20 exahertz to 5 exahertz.Alternatively, the electromagnetic sensor system may detect the band ofradiation represented by segments of a few consecutive bands, such as200 terahertz to 200 gigahertz, representing consecutive bands ofinfrared and microwave EMR. Alternatively, the electromagnetic sensorsystem may detect various different bands within different spectra, forinstance, 10 exahertz to 50 petahertz and 350 gigahertz to 350megahertz, the first band ranging from x-rays to soft x-rays, and thesecond band ranging from infrared to microwave EMR. The electromagneticsensor system may transmit data about the ambient electromagneticradiation to the tracking system 106 to be executed using instructionsstored in the memory system by the processor system.

In step 204, an embodiment of the electromagnetic sensor transmits thedata about the ambient electromagnetic radiation, including at least theparticular frequency band from the marker on the object, to a trackingsystem 106 (FIG. 1A).

In step 206, an embodiment of the processor system may be configured toexecute analysis and control routines 147 stored on the memory device todetermine whether any electromagnetic radiation detected includes atleast a particular frequency band. If the electromagnetic sensortransmits information about the particular frequency band, the processorwill use the data to find data that represents the marker at step 208,and/or determine by processor system the location of an object based onthe data representing the marker at step 210. If the electromagneticsensor does not detect the particular frequency band, theelectromagnetic sensor may continue to detect electromagnetic radiationusing an electromagnetic sensor at step 202.

In step 208, the processor system determines data representing themarker within the data about the ambient electromagnetic radiation. Byusing the sensor data from at least one sensor, the processor candetermine what part of the data contains the marker. In doing so, thetwo-dimensional, or three dimensional coordinate of the marker can besuperimposed on another set of image data to correlate the data tovisual images. A computer can track the location of a marker and if themarker is not visible, visually display where the marker is located. Themarker may be visible, invisible concealed for example, and therefore,the marker may appear more pronounced when the data from theelectromagnetic sensor is emphasized in the superimposed image.Alternatively or additionally, instead of superimposing images, themarker can be accentuated by limiting the electromagnetic radiation ofbands not reflected or emitted by the marker.

Detection software may have modules including any or all of apreprocessor module, a background subtraction module, an object detectormodule, an object tracker module or an object validation module.

The preprocessor module may preprocess the video data to make the datasuitable for further processing and analysis. The preprocessing involvessteps such as noise reduction and contrast enhancement.

The Background subtraction module may model the background conditions ofthe scene, thereby accentuating the non-background pixels in the sceneat the time of processing. The background model can be static or dynamicand adaptive. Static models are easier to compute, have lowcomputational complexity, but are applicable only in those cases wherescene conditions do not change with time. This assumption can berestrictive.

A variety of background modeling algorithms can be used for determiningthe background, such as a fixed-threshold approach, Gaussian mixturemodeling, kernel density estimators, mean-shift filtering, Kalmanfiltering, etc. The approach taken in this embodiment accounts for thefeedback from the Object Tracker module to determine which pixels in thescene are currently part of the foreground, and hence suppresses theupdating of the background model in the background regions.

In step 210, the processor system determines a location of an objectbased on the data representing the marker. The detection softwaremodules outlined above could have already refined this data. Thelocation of the object may be determined using a variety of methodsincluding a stereo image depth computation, three-dimensionalorientation computing, or object pose/orientation computation, forexample.

In method 200, each of the steps is a distinct step. In anotherembodiment, although depicted as distinct steps in FIG. 2, step 202-210may not be distinct steps. In other embodiments, the method to monitormay not have all of the above steps and/or may have other steps inaddition to or instead of those listed above. The steps of the method tomonitor may be performed in another order. Subsets of the steps listedabove as part of the method to monitor may be used to form their ownmethod.

FIG. 3 illustrates a flowchart of another embodiment of a method 300 ofmonitoring an object implemented by control and analysis routines 147,in which electromagnetic radiation can be used to help detect and locateobjects. In the embodiment of FIG. 3, the sensor system may detect aparticular band of electromagnetic radiation and use it to locate amarker on an object.

In optional step 302, the tracking system 106 instructs anelectromagnetic radiation emission device to emit electromagneticradiation (“EMR”) including at least a particular frequency band. Byadding a higher intensity of a particular band of electromagneticradiation, the marker may simply reflect the particular frequency bandthat may be detected to locate the object. Of course, in someenvironments with some spectra, there may be little need for an activetransmitter marker or an electromagnetic radiation emitter. Thereflection may very well be strong enough without the emission device ora filter may be used to at least partially filter out other wavelengthsnot reflected by the marker, and step 302 may be unnecessary.

In step 304, an electromagnetic sensor system may detect a particularband of EMR using at least one electromagnetic sensor, such as a cameraand/or a camera combined with one or more other sensor. Theelectromagnetic sensor system may detect electromagnetic radiation thatreflected off of the marker. The electromagnetic sensor may generate adata stream with data representing the measurement of ambient radiation.Detecting the electromagnetic radiation may be performed with a cameraand optionally with an array of sensors.

The electromagnetic sensor system may be used to detect any part of aband of EMR, any combination of bands of electromagnetic radiation, forexample. The EMR transmitted for the marker may be of many differentfrequencies. In an embodiment, the electromagnetic sensor senses anarrow band of radiation.

The electromagnetic sensor system may be used to detect the EMR band ofa particular type of EMR, for instance, by solely detecting the bandthat is 1, 5, 10 (for example), or 15 terahertz wide between 400terahertz to 790 terahertz, representing a narrow band of visual lightspectrum. Alternatively, the electromagnetic sensor system mayselectively detect a narrow band of the x-ray band, such as 20 exahertzto 5 exahertz. Alternatively, the electromagnetic sensor system maydetect a narrow band of radiation between 200 terahertz and 200gigahertz. Alternatively, the electromagnetic sensor system may detectvarious different bands within different spectra, for instance. Theelectromagnetic sensor system may transmit data about the ambientelectromagnetic radiation to the tracking system 106 to be analyzedusing instructions stored in the memory system by the processor system.

In step 306, an embodiment of the electromagnetic sensor transmits thedata about the ambient electromagnetic radiation, including the at leasta particular frequency band from the marker on the object, to a trackingsystem 106 (FIG. 1A).

In step 308, the tracking system 106 receives and analyzes the capturedimage data transmitted by a captured image device.

In step 310, an embodiment of the processor system may implementinstructions from the analysis and control routines 147 in order todetermine whether any electromagnetic radiation detected includes theparticular frequency band. The processor analyzes the data collectedfrom the electromagnetic sensor to find data that represents the markerat step 310 and/or determine by the processor system the location of anobject based on the data representing the marker at step 312. If theelectromagnetic sensor does not detect the particular frequency band,the electromagnetic sensor may continue to detect electromagneticradiation using an electromagnetic sensor at step 302.

In step 312, the processor system determines data representing themarker within the data about the ambient electromagnetic radiation. Byusing the sensor data from at least one sensor, the processor candetermine what part of the data contains the marker. In doing so, thetwo-dimensional, or three-dimensional coordinates of the marker can bedetermined. Tracking system 106 may track the location of a marker andvisually display where the marker is located.

In step 314, the processor system determines a location of an objectbased on the data representing the marker. The preprocessor 160, asdescribed in FIG. 1D, may have already refined the data representing themarker. The location of the object may be determined using a variety ofmethods including a stereo image depth computation, three-dimensionalorientation computing, object pose/orientation computation, for example.

In step 318, the tracking system 106 analyzes a direction of motion ofthe object based on comparing multiple image frames and transmitsinstructions to the image capture system to track the object byfollowing the marker. The camera (e.g., a pan tilt zoom camera) may beconfigured to track the object either by moving the image capturedevice, or by following the marker, digitally.

The tracking system 106 (FIG. 1A) may track objects moving from oneimage capture device's view to another image capture device's view.Tracking system 106 may detect where the object leaves the frame of oneimage capture device and where the object may return or enter the frameof another image capture device. Tracking the motion of an objectbetween frames may include determining the area where the object leftthe frame, determining a trajectory of motion of the abject whileleaving the frame, and detecting where the object reappears. Optionally,the tracking system 106 may implement instructions that cause trackingsystem 106 to send signals that cause an image capture device to awaitthe reentry of the object that left the view of a particular imagecapture device in order to detect the object. If for any reason, theobject disappears, the system may wait for the object to reappear.

The tracking system 106 (FIG. 1A) may command that a particulardetection and location image capture device signal another detection andlocation image capture device to track a particular object. In anembodiment, the processor system may be able to signal the exchange ofan object from the field of view of one tracking camera to the field ofview of another tracking camera. The cameras themselves may trigger oneanother to follow an object as well. Location and identification camerasmay be able to trigger tracking cameras, other location andidentification cameras, or other cameras. The triggered cameras couldalso trigger those cameras.

Optionally the tracking system 106 may receive instructions from aportable electronic device. For instance, the portable electronic devicemay be a cellular phone. The portable electronic device's processorsystem may be configured to execute instructions from an applicationstored on the portable electronic device memory system to perform any,all or none of the following functions: 1. Receive data from a server.2. Receive data from the tracking system 106, via the network interfacesystem. 3. Display data received on a display. 4. Transmit instructionsto the tracking system 106, transmits instructions to the server, andthe like, for example.

Optionally, the tracking system 106 may transmit the data to a server.The data can be any, all or none of data about the ambientelectromagnetic radiation including the at least a particular frequencyband, captured image data, data representing the marker within the dataabout the ambient electromagnetic radiation, data representing an imagein which the sensor readings are visually superimposed on capturedimages, data for tracking the movement of an object, for example.

In the embodiments of FIG. 3, each of the steps is a distinct step. Inanother embodiment, although depicted as distinct steps in FIG. 3, steps302-322 may not be distinct steps. In other embodiments, the method tomonitor may not have all of the above steps and/or may have other stepsin addition to or instead of those listed above. The steps of the methodto monitor may be performed in another order. Subsets of the stepslisted above as part of the method to monitor may be used to form theirown method.

FIGS. 4A-4C illustrate perspective views of embodiments of the marker.In FIG. 4A there is first image 402, there is a shirt 404 with a marker406. The marker 406 may be configured to reflect electromagneticradiation (“EMR”). The marker 406 may be of a material configured toreflect or emit a particular band of EMR, detected by an electromagneticsensor, such as a camera. Marker 406 is placed on the shirt of a personthat the tracking system 106 may track. Marker 406 is placed in thechest area of the shirt, but may be placed anywhere on the shirt, suchas the back and/or collar. In one particular embodiment, the shape ofthe marker 406 could be a rectangle. In other embodiments, the marker406 may include a triangle, square, hexagon, octagon, other polygon,circle, oval, and/or other shape, for example.

In FIG. 4B, image 408 shows another embodiment of the marker 406, whichcould be a shirt with a variety of EMR reflectors or emitters. Theembodiment of FIG. 4B multiple markers are placed on a shirt. Themarkers may include a first horizontal chest stripe 412 a, a secondhorizontal chest stripe 412 b, which may be parallel to the firsthorizontal chest stripe 412 a, a first horizontal arm stripe 414 a, anda second horizontal arm stripe 414 b, for example. Any design for thereflective or emissive material may suffice.

In FIG. 4C, image 416 shows another embodiment of the marker 406 inwhich marker 406 is placed on a hat 418. The hat 418 may be any hatincluding a baseball cap, a helmet, a top hat, or any other cap, forexample. Material configured to emit or reflect EMR can be placed aboutthe hat 418. There may be a front stripe 420, a first side stripe 422 a,a second side stripe 422 b, and/or a back stripe (not shown), forexample. In other embodiments, the stripes may be in other locations,such as the brim, the top, and/or the corners of the hat, in addition toand/or instead of the locations shown in FIG. 4C.

FIG. 5 illustrates a perspective view of an embodiment of anelectromagnetic sensor system. In FIG. 5, the electromagnetic sensorsystem 502 may include at least one sensor device 506 and at least oneimage capture device 504 (e.g., a camera).

The at least one image capture device 504 may be any device configuredto capture images. In this specification, the terms image capture deviceand camera are used interchangeably. The terms image capture device andcamera may be substituted with one another to obtain a differentembodiment. The at least one image capture device 504 may be used tocapture images in order to locate and identify objects.

The at least one sensor device 506 may include any number of sensorsconfigured to detect any, all, none, or some bands of electromagneticradiation (“EMR”). These sensors may be arranged in such a manner as towork in concert to detect and locate objects using a marker configuredto emit or reflect a particular band of EMR. The at least one sensordevice may detect the intensity of certain bands of EMR.

A processor system of tracking system 106 may be configured to implementinstructions found in applications stored in a memory system within thetracking system 106. The location of the object may be determined usinga variety of methods including a stereo image depth computation,three-dimensional orientation computing, object pose/orientationcomputation, for example. The tracking system 106 may also impose avisual representation of the electromagnetic radiation on an imagerepresented by the image capture data. The output may be image datarepresenting where a marker is located by showing, in a captured image,locations from where a particular band of radiation is reflected oremitted.

FIGS. 6A-6C illustrate a perspective view of an embodiment of a systemfor monitoring activity. Images 602 a, 602 b, and 602 c may show asimilarly positioned image at different aperture and/or filter settingsof an image capture device. An image capture device may apply a bandpass filter that partially filters out all wavelengths of light exceptfor a narrow band that characterizes the light emitted by the marker.Alternatively, the aperture may be partially closed, partially blockingall frequencies of light enhancing the contrast between the marker andthe rest of the image. An aperture stop may be any kind of stopincluding a diaphragm which expands and contracts to change the extentto which light is allowed to enter the image capture device. The shutterspeed of an image capture device can also affect the intensity of thelight presented in the resulting image. In FIGS. 6A-6C, the object hasno marker. FIGS. 6A-6C are shown to contrast with the ease ofidentifying the maker in FIGS. 7A-7C, in which the object has themarker.

In FIG. 6A, the first picture 602 a presents an image containing anobject captured by a camera with a shutter speed or frame exposure timeof 1/2000 of a second. The first object 604 a may be seen and detectedrelatively easily in the first picture 602 a.

In FIG. 6B, the second picture 602 b presents an image containing anobject captured by a camera with a shutter speed or frame exposure timeof 1/4000 of a second. The second object 604 b may still be seen, butthe brightness of the object may appear to fade relative to thebackground.

In FIG. 6C, the third picture 602 c presents an image containing anobject captured by a camera with shutter speed or frame exposure time of1/10000 of a second. The third object 604 c may be difficult to see atthis aperture setting, and the third object 604 c may be lessdistinguishable from its background.

FIGS. 7A-7C illustrate a perspective view of an embodiment of a systemfor monitoring. A marker can appear more pronounced when image data froman image capture device displays lower light intensity. Adjusting theaperture size (e.g., by changing the f-stop) or by changing the shutterspeed, or the amount of time the frame is exposed, may control the lightintensity. The aperture stop may be a diaphragm capable of adjusting theaperture size, for example. Shutter speed adjustment may also be used toreduce the intensity of light exposure a particular image receives inaddition to or instead of closing the aperture.

In FIG. 7A, picture 702 a presents an image with a first object 704 athat has a first marker 706 a on it, taken with a shutter speed or frameexposure time of 1/2000 of a second. The marker first 706 a may be thestripes on a striped shirt worn by the first object 704 a. The firstmarker 706 a may reflect or emit electromagnetic radiation (EMR) suchthat an electromagnetic sensor may detect the particular band or bandsof EMR the first marker 706 a reflects. By closing the aperture (orincreasing the shutter speed), the contrast of the marker relative tothe rest of the image is enhanced. The shutter speed or frame exposuretime setting of 1/2000 of a second darkens the background, so thehighlighted first marker 706 a may appear better illuminated whencompared with its background.

In FIG. 7B, picture 702 b presents an image with a second object 704 bthat has a second marker 706 b on it with a shutter speed or frameexposure time of 1/4000 of a second. The second object 704 b may be moredifficult to distinguish from the background, as the light intensity ofthe background is diminished. As such, the second marker 706 b mayappear more pronounced with a shutter speed or frame exposure time of1/4000 of a second than the first marker 706 a appeared with a shutterspeed or frame exposure time 1/2000 of a second.

In FIG. 7C, the third picture 702 c presents an image with a thirdobject 704 c that has a third marker 706 c on it, taken with a shutterspeed or frame exposure time 1/10000 of a second. The third object 704 cmay be almost indiscernible from the background. The third marker 706 cmay appear significantly more pronounced than the other elements in theshot. The markers 706 a, 706 b, and 706 c may not have significantlydimmed from picture to picture, despite the dimming of the rest of theimage as a result of a closing aperture stop or increasing the shutterspeed. The contrast created by closing the aperture and/or using ahigher shutter speed may allow the third marker 706 c to be the mostdistinguishable from its background.

FIGS. 8A-8C illustrate a perspective view of an embodiment of the systemfor monitoring. In FIGS. 8A-8C, rectangles 808 a, 808 b and 808 c havebeen placed around markers 806 a, 806 b and 806 c for comparison withthe markers. FIGS. 8A-8C are the same as FIGS. 7A-7C except thatrectangles have been drawn around the markers. A centroid of the regionof the marker may be computed to determine the location of the object.The shape of the marker compared to the rectangle may be used todetermine the orientation of the object.

Alternatives and Extensions

Each embodiment disclosed herein may be used or otherwise combined withany of the other embodiments disclosed. Any element of any embodimentmay be used in any embodiment.

Although the invention has been described with reference to specificembodiments, it will be understood by those skilled in the art thatvarious changes may be made and equivalents may be substituted forelements thereof without departing from the true spirit and scope of theinvention. In addition, modifications may be made without departing fromthe essential teachings of the invention.

What is claimed is:
 1. A method of monitoring, comprising: capturing,with an image capture device, a panoramic view; sensing, by anelectromagnetic sensor, ambient electromagnetic radiation, associatedwith the panoramic view, the ambient radiation including at leastdetecting a particular frequency band reflected from a marker on anobject, the electromagnetic sensor being at least one component of aportable electronic device; transmitting data about the ambientelectromagnetic radiation to a hardware system, the hardware systemincluding at least one memory system and a processor system having atleast one processor, the data that is transmitted including at least theparticular frequency band reflected from the marker on the object;determining, by the processor system of the hardware, data representingthe marker within the data about the ambient electromagnetic radiation;determining by the processor system of the hardware a location of theobject based on the data representing the marker; storing in the memorysystem the data about the ambient electromagnetic radiation from thesensor; storing in the memory system the data representing the markerwithin the data about the ambient electromagnetic radiation; storingdata generated by the processor regarding the location of the object,based on the data representing the marker location of the object; anddisplaying the data transmitted from the hardware system on the portableelectronic device or displaying the data transmitted from the hardwaresystem on another portable electronic device.
 2. The method of claim 1,further comprising causing an electromagnetic radiation source to emitthe particular frequency band of electromagnetic radiation, sending theparticular frequency band of electromagnetic radiation towards themarker, therein causing the marker to reflect the particular frequencyband to the sensor.
 3. The method of claim 1, the electromagnetic sensorbeing part of a moving image capture device, the method furthercomprising tracking motion of the marker by moving the moving imagecapture device.
 4. The method of claim 1, further comprising, trackingthe location of the object, with the electromagnetic sensor and thehardware system, from captured image data representing a field of viewof one image capture device, to captured image data representing a fieldof view of a second image capture device, as the object moves from thefield of view of one image capture device to the field of view of theother image capture device.
 5. The method of claim 4, the hardwaresystem further including a plurality of fixed image capture devices andmobile image capture devices, further comprising: instructing, by thehardware system, the moving image capture devices to follow the object;and instructing, by the hardware system, to display on a display animage consisting of a smaller range of view than that of a range of viewof the fixed image capture device.
 6. The method of claim 1, furthercomprising, transmitting by the hardware system an instruction to animage capture device to optically limit the extent to which the ambientelectromagnetic radiation may enter the image capture device, thereinincreasing the contrast between the marker and other parts of the image.7. The method of claim 1, further comprising determining, by thehardware system, an orientation of the marker based on comparing aperceived geometric shape of the marker represented by data transmittedfrom an image capture device and a known shape of the marker.
 8. Themethod of claim 1, further comprising: receiving, by the processorsystem, captured image data representing a first view of the marker froma first image capture device; receiving, by the processor system,captured image data representing a second view of the marker from asecond image capture device; determining, by the hardware system, theorientation of the marker based on the first view and the second view.9. A system for monitoring activity, comprising: an electromagneticsensor system, wherein the electromagnetic sensor system is at least onecomponent of a portable electronic device, the electromagnetic sensorsystem including at least one electromagnetic sensor configured to:detect ambient electromagnetic radiation, including at least aparticular frequency band reflected from a marker on an object; andtransmit data about the ambient electromagnetic radiation, including atleast the particular frequency band reflected from the marker on theobject, to a hardware system; the hardware system, including a processorsystem and a memory system, the processor system, including at least oneprocessor configured to: receive the data about the ambientelectromagnetic radiation including at least the particular frequencyband reflected from the marker on the object, from the sensor; determinea representation of the marker within the data about the ambientelectromagnetic radiation; and determine, a location of the object basedon the data representing the marker; and the memory system: storing thedata about the ambient electromagnetic radiation from the sensor;storing the data representing the marker within the data about theambient electromagnetic radiation; and storing data generated by theprocessor regarding the location of the object, based on the datarepresenting the marker location of the object; and the marker; and atleast one image capture device configured to offer a panoramic view;wherein the portable electronic device being configured to display datatransmitted from the hardware system or the system for monitoringactivity further including at least another portable electronic devicebeing configured to display data transmitted from the hardware system.10. The system of claim 9, wherein the marker transmits the at least aparticular frequency band of electromagnetic radiation.
 11. The systemof claim 9, wherein the marker selectively reflects at least theparticular frequency band of electromagnetic radiation.
 12. The systemof claim 9, further comprising at least one image capture systemincluding at least one image capture device configured to transmitcaptured image data to the hardware system.
 13. The system of claim 12,wherein at least one of the at least one image capture device is a movesand is configured to adjust a position of the image capture device sothat the location of the object based on the data representing themarker transmitted from the hardware system may be positioned at thecenter of an image represented by the captured image data.
 14. Thesystem of claim 9, the at least one image capture device that isconfigured to offer a panoramic view having a field of view spanningbetween 90 degrees and 360 degrees.
 15. The system of claim 9, furthercomprising a portable electronic device configured to communicate withthe hardware system.
 16. The system of claim 9, further comprising aportable electronic device including at least one image capture device,wherein the portable electronic device is configured to transmitcaptured image data to the hardware system.
 17. The system of claim 9,further comprising a server, wherein the hardware system is configuredto communicate with the server.
 18. The system of claim 9, the processorfurther configured to determine the whether a patient's activityrepresents a seizure based on instructions included in an applicationstored in the memory system, wherein the marker is coupled to a patientin order to track the level of activity of the patient.